Face reconstruction from monocular video using uncertainty analysis and a generic model

Reconstructing a 3Dmodel of a human face fromamonocular video sequence is an important problem in computer vision, with applications to recognition, surveillance, multimedia, etc. However, the quality of 3D reconstructions using structure frommotion (SfM) algorithms is often not satisfactory. One of the reasons is the poor quality of the input video data. Hence, it is important that 3D face reconstruction algorithms take into account the statistics representing the quality of the video. Also, because of the structural similarities of most faces, it is natural that the performance of these algorithms can be improved by using a generic model of a face. Most of the existing work using this approach initializes the reconstruction algorithm with this generic model. The problem with this approach is that the algorithm can converge to a solution very close to this initial value, resulting in a reconstruction which resembles the generic model rather than the particular face in the video which needs to be modeled. In this paper, we propose a method of 3D reconstruction of a human face from video in which the 3D reconstruction algorithm and the generic model are handled separately. We show that it is possible to obtain a reasonably good 3DSfM estimate purely from the video sequence, provided the quality of the input video is statistically assessed and incorporated into the algorithm. The final 3Dmodel is obtained after combining the SfM estimate and the generic model using an energy function that corrects for the errors in the estimate by comparing the local regions in the twomodels. The main advantage of our algorithm over others is that it is able to retain the specific features of the face in the video sequence even when these features are different from those of the generic model and it does so even as the quality of the input video varies. The evolution of the 3D model through the various stages of the algorithm and an analysis of its accuracy are presented. 2003 Published by Elsevier Inc. This work was partially supported by NSF Grant 086075 and DARPA/ONR Grant N00014-00-10908. * Corresponding author. Fax: 1-301-314-9115. E-mail addresses: [email protected] (A.K. Roy Chowdhury), [email protected] (R. Chellappa). URLs: http://www.cfar.umd.edu/~amitrc, http://www.cfar.umd.edu/~rama. 1077-3142/$ see front matter 2003 Published by Elsevier Inc. doi:10.1016/S1077-3142(03)00079-1 A.K. Roy Chowdhury, R. Chellappa / Computer Vision and Image Understanding 91 (2003) 188–213 189